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Evaluating Professional Development Schools: Challenges, Contradictions and 
Tensions. 



Abstract: 

This paper critically describes the difficult task of evaluating a Professional Development 
School (PDS), the often-contradictory opinions of the stakeholders in regard to acceptable 
conceptual models and the tensions therein. The description is a result of active participation in 
PDS related activities , document analysis and in-depth interviews with a select sample of PDS 
stakeholders in a small State of New York rural PDS initiative. The analysis proposes the 
development of a conceptual model that clearly establishes the desired outcomes through 
negotiation and consultation with all stakeholders , which ultimately informs the evaluation path. 
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Evaluating Professional Development Schools 

In 1998 State University of New York Oswego's School of Education, in collaboration 
with the nine school districts in Oswego County entered into a partnership to initiate a 
Professional Development School. This partnership jointly committed to achieve four main goals 
as espoused by Trachtman (1996): a) support student learning; b) enhance, reform and improve 
teacher education; c) provide professional development; and d) create positive educational 
change through inquiry. Professional Development Schools have been embraced by major 
professional organizations such as the American Association of Colleges for Teacher Education, 
and National Accreditation of Teacher Education (NCATE) as a way to renew K-12 schools as 
well as teacher education (Abdal-Haqq, 1998). The PDS initiative in the Oswego County has 
been piloted at two elementary school sites, and is in the process of being expanded to a middle 
school in the upcoming year. The long-range plans are to develop one PDS in each of the nine 
school districts, each with a unique focus and expertise. In January of 2000, 1 was designated to 
be the PDS internal evaluator on a 1 /4-time release from the School of Education. It was during 
this process that I found evaluating a PDS to be a difficult undertaking for a number of reasons. 
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The very description of Oswego County PDS as “learning communities who value inquiry, 
openness, inclusiveness, an appreciation of diversity, collaboration, and high standards for all 
students and faculty" (partnership agreement), is itself problematic. Schools and universities have 
different structural arrangements and cultural views that govern their existence. Synthesizing 
these two conceptually different institutions is challenging. Furthermore looking for impacts 
within a setting that is not really a PDS might taint the evaluation. Teitel ( 1 999) points out that it 
maybe too soon to measure impacts for these partnerships; “ they represent long-term systematic 
changes that should not be measured before the changes are in place” (p. 1 ). These relationships, 
which under gird collaborative partnerships, are fragile and can easily become damaged by a 
premature evaluation that demands instant results with little or no sensitivity towards nurturing 
the relationship. 

Secondly, pressure on those involved in PDS work to justify the considerable human and 
fiscal resources devoted to a PDS initiative provide a different set of challenges. Teitel (1999) 
reminds us that pressure often comes from state, district and higher education administrators who 
are asked to provide funds for PDS work. This pressure builds from skeptical school and 
university based teacher educators prodded to abandon established models of pre-service 
preparation to embrace new practices. It also comes from weary teachers asked to add mentoring, 
inquiry, and management tasks to their classroom duties, while cynical journalists, confused 
parents, curious researchers, and self-appointed guardians of tax payers interests put additional 
demands to an already pressured group to provide accountability (Teitel, 1999). 

Conceptual Framework : 

A recent NCATE Professional Development School study found several emerging themes 
characterizing those PDS schools. Based on a National survey of 28 PDS sites, the study found 
fundamental shifts in perspectives in the way teaching and learning is being done. Teachers no 
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longer hold all the information. As learners begin to take responsibility for knowledge 
construction, they are turning to one another or to new technologies to answer their questions. 
This constructivist approach implies new classroom settings where both teacher and students 
assume new roles. The students are expected to work independently on difficult tasks and to 
engage in inquiry-based projects and to use technology to create knowledge (Trachtman, 1998). 
Multiple approaches to assessment are being used to measure students learning, as the teachers 
become more concerned on how students learn and should be doing rather than what is learned. 
These educationally responsive schools view learners as active participants who take 
responsibility for their own learning (Abdal-Haqq, 1998). More avenues are being created for 
teachers to develop new skills and opportunities are being provided to learn with others. 
Teaching is being viewed as a collaborative, teamwork and interdependent activity. Group work 
and research teams are appearing at these sites, as teachers feel safe to try and fail in a supported 
environment. 

The learning community of teachers is beginning to raise specific questions about practice 
and seeking ways to answer those questions systematically. This enables the teachers to use the 
findings to inform practice and to share their findings in formal situations with others who have 
similar questions and concerns (Trachtman, 1998). Teachers in these sites have become doers and 
consumers of research. This in itself has become an avenue in which teacher voice is being 
privileged (Abdal-Haqq, 1998). 

The challenging task for evaluation is to measure how these changes impact student 
learning and specifically, achievement. PDS evaluation has the difficult challenge of informing 
those who do PDS work what PDS means to student outcomes. The need for an acceptable 
conceptual model that is agreeable to all stakeholders cannot be underscored. One in which the 
desired outcomes will be established and evaluation strategies that measure those desired 
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outcomes become the guiding factor. 

Methodology 

Collection of data for PDS evaluation has occurred from different sources and multiple locations: 

a) In-depth interviews 

b) Document analysis 

c) Participatory observations 

In-depth interviews: 

Joining PDS long after it was established, it was necessary that I piece together the various 
elements that had brought it into existence. An important part would have to come from the 
people who had been closely involved with the establishment of the PDS. To this end, I chose to 
talk with and interview 15 people who were original participants of the PDS. This included 6 
public school-based faculty, 4 university-based faculty, 2 public school principals, 1 
superintendent, 1 Dean and the PDS co-ordinator. These interviews that took the form of 
conversational discussions varied from one individual to the other, and how much they were 
willing or able to discuss and share. The time range was 30 minutes to 2 hours. All the interviews 
were audiotaped and later transcribed. 
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Docwnent analysis: 

Needless to say, a project of this magnitude has generated mountains of documents. There are 
three levels of administrative structure. At one end is the joint administrative group that steers the 
PDS work, otherwise known as the Team Sheldon. The group is comprised of the nine Oswego 
County school district superintendents, the Dean of school of education at Oswego State 
University, Associate Dean, and 5 department chairs. All the members of faculty directly 
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involved in the PDS work and the PDS co-ordinator are invited to the meetings to give updates 
and reports. This team meets once a month during the school year. I have been able to access the 
minutes of the meetings as well as much of the documentation that has been generated during 
these meetings. 

Another level of administration structure is the PDS council. This council that is 
comprised of school and university-based faculty, principals and 2 superintendents is charged 
with the responsibility of developing the goals and objectives .of the PDS. It is supposed to solicit 
views from those directly involved with PDS work and to develop policy. The council meets 
once a month. 

The third level of administration is the team management. Each school has a management 
team that comprises of school and university-based teachers, school principal and occasionally, 
parents. This team is charged with the responsibility of implementing policy and making the day- 
to-day plans of the site. The management teams meet several times a month. 

The PDS co-ordinator works back and forth in these 3 levels. She has generated a lot of 
paper work and documents, including a newsletter that goes out to the community once a month. 
This newsletter breaks down the activities that has take place during the month. Document 
analysis as a way of collecting data has generated a lot of important information. 

Participatory observations 

As the evaluator, I have access to all the activities and meetings related to PDS. During these 
activities, I gather a lot of information through the discussion that take place giving me a deeper 
understanding of what is going on. Some of the issues that are hard to capture in the written 
documents are important sources of information. 
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Evaluating Oswego County PDS: Challenges, Contradictions and Tensions. 

The challenge : 

It was apparent right at the moment I joined PDS as an evaluator that I was going to face a lot of 

difficulties and challenges. As far as I know, there was no established baseline prior to the 

implementation of the PDS. Team Sheldon had made the decision to implement a PDS with the 

help of a grant (Targeted Instructional Professional Development), a New York State Education 

Grant. According to the Dean of School of education, Team Sheldon has desired over time to 

have a joint long-term activity committed to providing leadership, collaboration and research. 

PDS seemed to be the obvious avenue where such a goal would be realized. Consequently, a 

consultant working with NCATE was invited during that summer to work with a group of 40 

people from pubic schools and university in a 2- day working retreat. 

And the retreat was in July. So, when we got there, she (consultant) gave us some 
overview of the critical components of a PDS. And how much it is a learning 
community. Everyone needs to participate as a member; no one is a guest, on 
equal footing. (Administrator, interview transcript) 

What this meant to each individual at that retreat is hard to tell. But it is clear they came out with 

a commitment to implement a PDS. 

We certainly did not know as a faculty what this would be. Nine school districts, 
the BOCES centre and the university coming together to talk. Not that we were 
information based. We all left thinking, what parts do we visualize working with 
given that we have a university in the rural area with 9 rural based districts and 
one small city. How can we make this work? (Administrator, interview transcript) 

Amidst all this uncertainty, a group came together to talk about the possibility of writing a grant 

that would establish a base to pilot a PDS initiative that year. 

So, it was a very aggressive proposal. US$300,000.. and we got it. And the group 
said they wanted to go forward with it. And so they commissioned a small 
committee to get together in January of 1 999 to set up a budget for what it would 
take to pilot (administrator, interview transcript). 
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The designated evaluator at the time the grant was written was no longer a participant in the PDS. 
As a new person, it was hard to establish the reasons why this person had left this role and made 
no effort to hand over when I came on board. The official explanation was that the person was 
overseeing a couple of other grants. However, little was explained as to why there was no 
documentation showing the evaluation activities that had taken place. Everyone seemed to be 
relieved that there is now a designated evaluator and everything would be fine. 

Secondly, the PDS was implemented without a clear criterion. With the grant in hand and 
no prior planning, two faculty members who had established some kind of relationship with 
public schools were hastily designated as the PDS liaisons. They were released for a 14 time from 
their previous duties and assigned to work for at least once a week in the schools, establishing 
some kind of a partnership. 

I think CW (colleague) did it to me. I was not at the original retreat a couple of 
years ago when the consultant came in. There was a lot of talk about PDS like 
activities going on and so on. The next thing I knew is that I was invited to a 
Team Sheldon meeting and the topic came up . . . how about L. as a site for PDS? 
(Faculty member, interview transcript) 

While the NCATE standards have been used as a benchmark, the stakeholders were not quite 
sure at the onset what the goals were and nothing was in place to show how those goals would be 
met. 



The council is set-up to broadly oversee the day-to-day operations of both sides. 
Two meetings ago, B. suggested that we really need to do some goal setting: 
where do want to be in June, where do we want to be in five years. Then what are 
the outcomes that will let us know we are there? Once we have those outcomes 
framed then the evaluation should grow out of that. (Faculty member, interview 
transcript). 




This conversation took place a few weeks after I was on as the evaluator. As such, I was put in a 
“sink or swim “ position, to figure out on my own where this broadly represented group wanted 
to go and then measure whether they have gone there. 
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The contradictions: 



In any PDS, there are two main players : The university and the public schools. These two 
institutions have different cultures and structures. Evaluating them as a cohesive group created a 
lot of contradictions. The university stakeholders are often concerned with the preparation of 
student teachers. The thrust recently has been to prepare teachers who can work in diverse 
situations, following the state mandate that all teacher candidates must be placed on an urban 
diverse situations at least on part of their student teaching. For rural districts, this is a huge 
contradiction because they would like to retain the teachers in the rural areas so that they can 
have a larger resource pool from which to hire. The public school on the other hand is 
concerned with academic achievements ofK-12. Regularly, the superintendents wanted to know 
what this partnership meant to achievement scores. How is the learning of K- 12 affected? Each 
of these people wants to see instant results. By working together, the superintendents were eager 
to see improved scores in state exams. The university is more concerned with producing teachers 
who are "socially conscious catalysts for change who create and sustain school environments 
where excellence is cherished and social justice flourishes" (SOE Conceptual Framework, 1998) 
which may sometimes include challenging the notion of standardized tests. Getting this 
information had to be couched with language that is acceptable to both. When reporting the 
changes in the recent standardized tests, I cautioned the group not to read too much in the data I 
presented for a number of reasons: 

a) Using standardised, group administered achievement scores to measure the impact of 
partner schoolwork on student learning is problematic. Merely comparing aggregated 
student scores across years and looking for evidence of change in those scores is a curious 
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task . So many variables impact a school in any given year that connecting partner work to 
gains or losses in achievement data is at best a limited relationship . 

b) Leadership and staff changes ultimately affect school outcomes far more strongly than 
partner linkage. There are a variety of other changes that can potentially alter school 
performance such as changes in the local economy , demographic changes in the community \ 
the adoption of new curriculum or textbooks , teacher changes , the number of students at any 
grade level etc. 

c) Literature on student performance as measured through standardised test scores suggests 
that it be only after a high quality implementation over an extended period of time (5-7 years) 
would there be some trends noticeable. 

What I would suggest is to use this data to inform professional development activities for school 
and university based faculty and leaders. For example , which sub-populations are not meeting 
the standards and what should we do to move everyone to the acceptable levels? 

The tension: 

It is hard enough to evaluate a project with specific goals; it is harder to evaluate one with no 
such goals. I found myself working with a group of tense people, not sure what was being 
evaluated. Often, people would confide, off the record, about the frustrations of doing something 
they were quite unsure of or something they thought might raise some ethical issues. For the first 
time, the landscape was changing and there was nothing to hold on to. Then there was the issue 
of resources. 

While a lot of these children have serious emotional problem which are causing 
this behaviour. And unfortunately because of the resources in the school, we 
don’t know who is going to continue working with them. So, I guess the question 
in my mind is giving a child two weeks of assistance and support that is enough 
for us or are we opening the Pandora's box by dealing with children and opening 

10 



up these concerns, and then not having enough time for a follow up. (Faculty 
member, interview transcript) 

The tension arose because some individuals who made some commitment to participate in the 

initiative had left the institution. Anyone who was made to fit in their shoes was unable to 

interpret the expectations and responsibilities associated with the role. 

We had one faculty member. She is not with us anymore that was involved with 
some initial meetings and then she left. And she appeared to have made some 
commitment that our department would be involved in some way. (Faculty 
member, interview transcript) 

Assuming a new role in a short period of time has the potential of making some participants tense 

particularly when they are unsure of what they are expected to do: 

J. (head of department) asked me to give him some ideas as to how our 
involvement to the PDS might be. And I said I’m not sure I have any ideas. But 
somehow, this idea of putting the practicum out into the schools seemed to be a 
good one (Faculty member, interview transcript). 

The need for data surpassed the supply, thanks to the resources set aside for evaluation that were 

not readily available and which ultimately caused some tension. All this data collection and 

analysis was supposed to be done and reported to all the stakeholders on a 1/4 time, which 

amounts to 10 hours a week. There was the constant push from all stakeholders to be told how 

they were doing. Even though they were not themselves sure what they ‘ought’ to be doing. 

Every time I appeared in the building, people would ask what it is I was evaluating . . . this often 

created some tension because I felt like I was policing. Other people invited me to come and see 

some “wonderful” projects they were doing at their sites. It looked more or less like my work 

would be to give assurance that all is well. 



Developing an effective evaluation strategy 

PDS has the potential of effecting much needed reforms in both higher education and public 



schools. Each member of the PDS will work towards a better understanding of the relationship 
between educational theories and educational practices in an effort to improve instruction for all 
children. For this to happen, an effective evaluation strategy has to be put into place. This will 
involve going back to the drawing board and establishing clear goals, making public those goals, 
and devising a structure that will help meet those goals. Because PDS is a broad undertaking, 
more than one individual is required to assess the impacts and specifically, whether it is 
promoting students learning. I envision an on going self-study process where stakeholders will be 
in control of assessing their own work. All participants of the PDS initiative at various roles may 
want to take sometime to look at their accomplishments in relation to the goals they set up for 
themselves. And with the desire to renew and improve the practices that may be hindering the 
desired accomplishments. Whether there will be resources to support this structure is beyond the 
scope of this paper. 
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